Automated endoscope calibration

ABSTRACT

A surgical robotic system automatically calibrates tubular and flexible surgical tools such as endoscopes. By accounting for nonlinear behavior of an endoscope, the surgical robotic system can accurately model motions of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. The surgical robotic system models the nonlinearities using sets of calibration parameters determined based on images captured by an image sensor of the endoscope. Calibration parameters can describe translational or rotational movements of the endoscope in one or more axis, e.g., pitch and yaw, as well as a slope, hysteresis, or dead zone value corresponding to the endoscope&#39;s motion. The endoscope can include tubular components referred to as a sheath and leader. An instrument device manipulator of the surgical robotic system actuates pull wires coupled to the sheath or the leader, which causes the endoscope to articulate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.15/191,391, filed Jun. 23, 2016, which claims the benefit of andpriority to U.S. Provisional Application No. 62/185,135, filed Jun. 26,2015, each of which is incorporated by reference herein in its entirety.The subject matter of the present application is related to U.S.application Ser. No. 14/523,760, filed on Oct. 24, 2014, entitled“SYSTEM FOR ROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS,”the entire disclosure of which is incorporated herein by reference.

BACKGROUND 1. Field of Art

This description generally relates to surgical robotics, andparticularly to an automated process for calibrating endoscopes.

2. Description of the Related Art

Robotic technologies have a range of applications. In particular,robotic arms help complete tasks that a human would normally perform.For example, factories use robotic arms to manufacture automobiles andconsumer electronics products. Additionally, scientific facilities userobotic arms to automate laboratory procedures such as transportingmicroplates. Recently, physicians have started using robotic arms tohelp perform surgical procedures. For instance, physicians use roboticarms to control surgical instruments such as endoscopes.

Endoscopes with movable tips help perform surgical procedures in aminimally invasive manner. A movable tip can be directed to a remotelocation of a patient, such as the lung or blood vessel. Deviation ofthe tip's actual position from a target position may result inadditional manipulation to correct the tip's position. Incorporatingreal time feedback of endoscope motions is difficult, for example,because endoscope tips are compressible and have a hysteresis. Further,existing techniques for manual calibration may rely on limited amountsof endoscope tip deflection that does not accurately model motions ofthe tip.

SUMMARY

A surgical robotic system automatically calibrates tubular and flexiblesurgical tools such as endoscopes. Surgical tools may exhibit nonlinearbehavior, for example, due to friction and stiffness of the tool'smaterial. By accounting for nonlinear behavior of an endoscope, thesurgical robotic system can accurately model motions of the endoscopeand navigate the endoscope while performing a surgical procedure on apatient. The surgical robotic system models the nonlinear behavior andmovements using sets of calibration parameters determined based onimages captured by an image sensor of the endoscope.

Calibration parameters can be determined using an image registrationprocess. Changes between two of the captured images correspond to ashift in perspective of the image sensor due to a movement of theendoscope. For instance, the endoscope moves along a trajectory inside acalibration structure while capturing images of the surface of thecalibration structure. The surgical robotic system calculates differencearrays and gradient arrays based on processing the captured images.Calibration parameters based on the arrays describe translational orrotational movements of the endoscope in one or more axis, e.g., pitchand yaw.

Calibration parameters can also be determined using calibration curves.The surgical robotic system generates the calibration curves based onposition and orientation information of the endoscope captured bysensors. Calibration parameters based on the calibration curves describea slope, hysteresis, or a dead zone value corresponding to theendoscope's movement in one or more axis.

In some embodiments, an endoscope includes tubular components referredto as a sheath and leader. The surgical robotic system moves the sheathand leader using an instrument device manipulator (IDM). For example,the IDM actuates pull wires coupled to the sheath or the leader, whichcauses the endoscope to articulate along different axis. The pull wiresmay also exhibit nonlinear behavior that can be modeled using thecalibration parameters. The sheath and leader may include a helixsection to mitigate unwanted bending and torqueing forces in theendoscope.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a surgical robotic system according to oneembodiment.

FIG. 2 illustrates a command console for a surgical robotic systemaccording to one embodiment.

FIG. 3A illustrates multiple degrees of motion of an endoscope accordingto one embodiment.

FIG. 3B is a top view of an endoscope according to one embodiment.

FIG. 3C is a cross sectional side view of a sheath of an endoscopeaccording to one embodiment.

FIG. 3D is an isometric view of a helix section of a sheath of anendoscope according to one embodiment.

FIG. 3E is another isometric view of a helix section of a sheath of anendoscope according to one embodiment.

FIG. 3F is a side view of a sheath of an endoscope with a helix sectionaccording to one embodiment.

FIG. 3G is another view of the sheath of the endoscope shown in FIG. 3Faccording to one embodiment.

FIG. 3H is a cross sectional side view of a leader of an endoscopeaccording to one embodiment.

FIG. 3I is a cross sectional isometric view of the leader of theendoscope shown in FIG. 3H according to one embodiment.

FIG. 4A is an isometric view of an instrument device manipulator of asurgical robotic system according to one embodiment.

FIG. 4B is an exploded isometric view of the instrument devicemanipulator shown in FIG. 4A according to one embodiment.

FIG. 4C is an isometric view of an independent drive mechanism of theinstrument device manipulator shown in FIG. 4A according to oneembodiment.

FIG. 4D illustrates a conceptual diagram that shows how forces may bemeasured by a strain gauge of the independent drive mechanism shown inFIG. 4C according to one embodiment.

FIG. 5A illustrates an example calibration setup for arobotically-controlled endoscope according to one embodiment.

FIG. 5B illustrates the endoscope positioned within a calibrationstructure according to one embodiment.

FIG. 6A shows a plot of measured endoscope actual deflection in pitchand yaw in response to a calibration procedure according to oneembodiment.

FIG. 6B shows a plot of linear curves corresponding to increasing targetdeflection in the pitch axis according to one embodiment.

FIG. 6C shows a plot of linear curves corresponding to decreasing targetdeflection in the pitch axis according to one embodiment.

FIG. 6D shows a plot of linear curves corresponding to increasing targetdeflection in the yaw axis according to one embodiment.

FIG. 6E shows a plot of linear curves corresponding to decreasing targetdeflection in the yaw axis according to one embodiment.

FIG. 7 is a flowchart of a process that may be performed as part of theprocess illustrated in FIG. 8 to determine the movements of theendoscope from a sequence of recorded images according to oneembodiment.

FIG. 8 is a flowchart of a process for automated calibration of anendoscope according to one embodiment.

FIG. 9 is a flowchart of a process for controlling an endoscope usingcalibration parameters, according to one embodiment.

FIG. 10A illustrates the distal end of an endoscope within an anatomicallumen according to one embodiment.

FIG. 10B illustrates the endoscope shown in FIG. 10A in use at anoperative site according to one embodiment.

FIG. 10C illustrates the endoscope shown in FIG. 10B with an aspirationneedle according to one embodiment.

FIG. 11A illustrates an endoscope coupled to a distal flexure sectionwithin an anatomical lumen according to one embodiment.

FIG. 11B illustrates the endoscope shown in FIG. 11A with a forceps toolin use at an operative site according to one embodiment.

FIG. 11C illustrates the endoscope shown in FIG. 11A with a laser devicein use at an operative site according to one embodiment.

The figures depict embodiments of the present invention for purposes ofillustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION

The methods and apparatus disclosed herein are well suited for use withone or more endoscope components or steps as described in U.S.application Ser. No. 14/523,760, filed on Oct. 24, 2014, published asU.S. Pat. Pub. No. US 2015/0119637, entitled “SYSTEM FORROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS,” the fulldisclosure of which has been previously incorporated by reference. Theaforementioned application describes system components, endolumenalsystems, virtual rail configurations, mechanism changer interfaces,instrument device manipulators (IDMs), endoscope tool designs, controlconsoles, endoscopes, instrument device manipulators, endolumenalnavigation, and endolumenal procedures suitable for combination inaccordance with embodiments disclosed herein.

I. Surgical Robotic System

FIG. 1 illustrates a surgical robotic system 100 according to oneembodiment. The surgical robotic system 100 includes a base 101 coupledto one or more robotic arms, e.g., robotic arm 102. The base 101 iscommunicatively coupled to a command console, which is further describedwith reference to FIG. 2 in Section II. Command Console. The base 101can be positioned such that the robotic arm 102 has access to perform asurgical procedure on a patient, while a user such as a physician maycontrol the surgical robotic system 100 from the comfort of the commandconsole. In some embodiments, the base 101 may be coupled to a surgicaloperating table or bed for supporting the patient. Though not shown inFIG. 1 for purposes of clarity, the base 101 may include subsystems suchas control electronics, pneumatics, power sources, optical sources, andthe like. The robotic arm 102 includes multiple arm segments 110 coupledat joints 111, which provides the robotic arm 102 multiple degrees offreedom, e.g., seven degrees of freedom corresponding to seven armsegments. The base 101 may contain a source of power 112, pneumaticpressure 113, and control and sensor electronics 114—includingcomponents such as a central processing unit, data bus, controlcircuitry, and memory—and related actuators such as motors to move therobotic arm 102. The electronics 114 in the base 101 may also processand transmit control signals communicated from the command console.

In some embodiments, the base 101 includes wheels 115 to transport thesurgical robotic system 100. Mobility of the surgical robotic system 100helps accommodate space constraints in a surgical operating room as wellas facilitate appropriate positioning and movement of surgicalequipment. Further, the mobility allows the robotic arms 102 to beconfigured such that the robotic arms 102 do not interfere with thepatient, physician, anesthesiologist, or any other equipment. Duringprocedures, a user may control the robotic arms 102 using controldevices such as the command console.

In some embodiments, the robotic arm 102 includes set up joints that usea combination of brakes and counter-balances to maintain a position ofthe robotic arm 102. The counter-balances may include gas springs orcoil springs. The brakes, e.g., fail safe brakes, may be includemechanical and/or electrical components. Further, the robotic arms 102may be gravity-assisted passive support type robotic arms.

Each robotic arm 102 may be coupled to an instrument device manipulator(IDM) 117 using a mechanism changer interface (MCI) 116. The IDM 117 canbe removed and replaced with a different type of IDM, for example, afirst type of IDM manipulates an endoscope, while a second type of IDMmanipulates a laparoscope. The MCI 116 includes connectors to transferpneumatic pressure, electrical power, electrical signals, and opticalsignals from the robotic arm 102 to the IDM 117. The MCI 116 can be aset screw or base plate connector. The IDM 117 manipulates surgicalinstruments such as the endoscope 118 using techniques including directdrive, harmonic drive, geared drives, belts and pulleys, magneticdrives, and the like. The MCI 116 is interchangeable based on the typeof IDM 117 and can be customized for a certain type of surgicalprocedure. The robotic 102 arm can include a joint level torque sensingand a wrist at a distal end, such as the KUKA AG® LBR5 robotic arm.

The endoscope 118 is a tubular and flexible surgical instrument that isinserted into the anatomy of a patient to capture images of the anatomy(e.g., body tissue). In particular, the endoscope 118 includes one ormore imaging devices (e.g., cameras or sensors) that capture the images.The imaging devices may include one or more optical components such asan optical fiber, fiber array, or lens. The optical components movealong with the tip of the endoscope 118 such that movement of the tip ofthe endoscope 118 results in changes to the images captured by theimaging devices. The endoscope 118 is further described with referenceto FIGS. 3A-I in Section III. Endoscope.

Robotic arms 102 of the surgical robotic system 100 manipulate theendoscope 118 using elongate movement members. The elongate movementmembers may include pull wires, also referred to as pull or push wires,cables, fibers, or flexible shafts. For example, the robotic arms 102actuate multiple pull wires coupled to the endoscope 118 to deflect thetip of the endoscope 118. The pull wires may include both metallic andnon-metallic materials such as stainless steel, Kevlar, tungsten, carbonfiber, and the like. The endoscope 118 may exhibit nonlinear behavior inresponse to forces applied by the elongate movement members. Thenonlinear behavior may be based on stiffness and compressibility of theendoscope 118, as well as variability in slack or stiffness betweendifferent elongate movement members.

The surgical robotic system 100 includes a controller 120, for example,a computer processor. The controller 120 includes a calibration module125, image registration module 130, and a calibration store 135. Thecalibration module 125 can characterize the nonlinear behavior using amodel with piecewise linear responses along with parameters such asslopes, hystereses, and dead zone values. The calibration module 125 andcalibration store 135 are further described in Sections IV-V:Calibration Dome and Calibration Curves. The surgical robotic system 100can more accurately control an endoscope 118 by determining accuratevalues of the parameters. The surgical robotic system 100 also uses theimage registration module 130 for calibration, which is furtherdescribed in Section VI. Image Registration. In some embodiments, someor all functionality of the controller 120 is performed outside thesurgical robotic system 100, for example, on another computer system orserver communicatively coupled to the surgical robotic system 100.

II. Command Console

FIG. 2 illustrates a command console 200 for a surgical robotic system100 according to one embodiment. The command console 200 includes aconsole base 201, display modules 202, e.g., monitors, and controlmodules, e.g., a keyboard 203 and joystick 204. In some embodiments, oneor more of the command module 200 functionality may be integrated into abase 101 of the surgical robotic system 100 or another systemcommunicatively coupled to the surgical robotic system 100. A user 205,e.g., a physician, remotely controls the surgical robotic system 100from an ergonomic position using the command console 200.

The console base 201 may include a central processing unit, a memoryunit, a data bus, and associated data communication ports that areresponsible for interpreting and processing signals such as cameraimagery and tracking sensor data, e.g., from the endoscope 118 shown inFIG. 1. In some embodiments, both the console base 201 and the base 101perform signal processing for load-balancing. The console base 201 mayalso process commands and instructions provided by the user 205 throughthe control modules 203 and 204. In addition to the keyboard 203 andjoystick 204 shown in FIG. 2, the control modules may include otherdevices, for example, computer mice, trackpads, trackballs, controlpads, video game controllers, and sensors (e.g., motion sensors orcameras) that capture hand gestures and finger gestures.

The user 205 can control a surgical instrument such as the endoscope 118using the command console 200 in a velocity mode or position controlmode. In velocity mode, the user 205 directly controls pitch and yawmotion of a distal end of the endoscope 118 based on direct manualcontrol using the control modules. For example, movement on the joystick204 may be mapped to yaw and pitch movement in the distal end of theendoscope 118. The joystick 204 can provide haptic feedback to the user205. For example, the joystick 204 vibrates to indicate that theendoscope 118 cannot further translate or rotate in a certain direction.The command console 200 can also provide visual feedback (e.g., pop-upmessages) and/or audio feedback (e.g., beeping) to indicate that theendoscope 118 has reached maximum translation or rotation.

In position control mode, the command console 200 uses athree-dimensional (3D) map of a patient and pre-determined computermodels of the patient to control a surgical instrument, e.g., theendoscope 118. The command console 200 provides control signals torobotic arms 102 of the surgical robotic system 100 to manipulate theendoscope 118 to a target location. Due to the reliance on the 3D map,position control mode requires accurate mapping of the anatomy of thepatient.

In some embodiments, users 205 can manually manipulate robotic arms 102of the surgical robotic system 100 without using the command console200. During setup in a surgical operating room, the users 205 may movethe robotic arms 102, endoscopes 118, and other surgical equipment toaccess a patient. The surgical robotic system 100 may rely on forcefeedback and inertia control from the users 205 to determine appropriateconfiguration of the robotic arms 102 and equipment.

The display modules 202 may include electronic monitors, virtual realityviewing devices, e.g., goggles or glasses, and/or other means of displaydevices. In some embodiments, the display modules 202 are integratedwith the control modules, for example, as a tablet device with atouchscreen. Further, the user 205 can both view data and input commandsto the surgical robotic system 100 using the integrated display modules202 and control modules.

The display modules 202 can display 3D images using a stereoscopicdevice, e.g., a visor or goggle. The 3D images provide an “endo view”(i.e., endoscopic view), which is a computer 3D model illustrating theanatomy of a patient. The “endo view” provides a virtual environment ofthe patient's interior and an expected location of an endoscope 118inside the patient. A user 205 compares the “endo view” model to actualimages captured by a camera to help mentally orient and confirm that theendoscope 118 is in the correct—or approximately correct—location withinthe patient. The “endo view” provides information about anatomicalstructures, e.g., the shape of an intestine or colon of the patient,around the distal end of the endoscope 118. The display modules 202 cansimultaneously display the 3D model and computerized tomography (CT)scans of the anatomy the around distal end of the endoscope 118.Further, the display modules 202 may overlay pre-determined optimalnavigation paths of the endoscope 118 on the 3D model and CT scans.

In some embodiments, a model of the endoscope 118 is displayed with the3D models to help indicate a status of a surgical procedure. Forexample, the CT scans identify a lesion in the anatomy where a biopsymay be necessary. During operation, the display modules 202 may show areference image captured by the endoscope 118 corresponding to thecurrent location of the endoscope 118. The display modules 202 mayautomatically display different views of the model of the endoscope 118depending on user settings and a particular surgical procedure. Forexample, the display modules 202 show an overhead fluoroscopic view ofthe endoscope 118 during a navigation step as the endoscope 118approaches an operative region of a patient.

III. Endoscope

FIG. 3A illustrates multiple degrees of motion of an endoscope 118according to one embodiment. The endoscope 118 is an embodiment of theendoscope 118 shown in FIG. 1. As shown in FIG. 3A, the tip 301 of theendoscope 118 is oriented with zero deflection relative to alongitudinal axis 306 (also referred to as a roll axis 306). To captureimages at different orientations of the tip 301, a surgical roboticsystem 100 deflects the tip 301 on a positive yaw axis 302, negative yawaxis 303, positive pitch axis 304, negative pitch axis 305, or roll axis306. The tip 301 or body 310 of the endoscope 118 may be elongated ortranslated in the longitudinal axis 306, x-axis 308, or y-axis 309.

The endoscope 118 includes a reference structure 307 to calibrate theposition of the endoscope 118. For example, the surgical robotic system100 measures deflection of the endoscope 118 relative to the referencestructure 307. The reference structure 307 is located on a proximal endof the endoscope 118 and may include a key, slot, or flange. Thereference structure 307 is coupled to a first drive mechanism forcalibration and coupled to a second drive mechanism, e.g., the IDM 117,to perform a surgical procedure. The calibration process of theendoscope is further described in Sections IV-VII. Calibration Dome,Calibration Curves, Image Registration, and Process Flows.

FIG. 3B is a top view of an endoscope 118 according to one embodiment.The endoscope 118 includes a leader 315 tubular component nested orpartially nested inside and longitudinally-aligned with a sheath 311tubular component. The sheath 311 includes a proximal sheath section 312and distal sheath section 313. The leader 315 has a smaller outerdiameter than the sheath 311 and includes a proximal leader section 316and distal leader section 317. The sheath base 314 and the leader base318 actuate the distal sheath section 313 and the distal leader section317, respectively, for example, based on control signals from a user ofa surgical robotic system 100. The sheath base 314 and the leader base318 are, e.g., part of the IDM 117 shown in FIG. 1.

Both the sheath base 314 and the leader base 318 include drivemechanisms (e.g., the independent drive mechanism further described withreference to FIG. 4A-D in Section III. D. Instrument Device Manipulator)to control pull wires coupled to the sheath 311 and leader 315. Forexample, the sheath base 314 generates tensile loads on pull wirescoupled to the sheath 311 to deflect the distal sheath section 313.Similarly, the leader base 318 generates tensile loads on pull wirescoupled to the leader 315 to deflect the distal leader section 317. Boththe sheath base 314 and leader base 318 may also include couplings forthe routing of pneumatic pressure, electrical power, electrical signals,or optical signals from IDMs to the sheath 311 and leader 314,respectively. A pull wire may include a steel coil pipe along the lengthof the pull wire within the sheath 311 or the leader 315, whichtransfers axial compression back to the origin of the load, e.g., thesheath base 314 or the leader base 318, respectively.

The endoscope 118 can navigate the anatomy of a patient with ease due tothe multiple degrees of freedom provided by pull wires coupled to thesheath 311 and the leader 315. For example, four or more pull wires maybe used in either the sheath 311 and/or the leader 315, providing eightor more degrees of freedom. In other embodiments, up to three pull wiresmay be used, providing up to six degrees of freedom. The sheath 311 andleader 315 may be rotated up to 360 degrees along a longitudinal axis306, providing more degrees of motion. The combination of rotationalangles and multiple degrees of freedom provides a user of the surgicalrobotic system 100 with a user friendly and instinctive control of theendoscope 118.

III. A. Endoscope Sheath

FIG. 3C is a cross sectional side view of the sheath 311 of theendoscope 118 according to one embodiment. The sheath 311 includes alumen 323 sized to accommodate a tubular component such as the leader315 shown in FIG. 3B. The sheath 311 includes walls 324 with pull wires325 and 326 running through conduits 327 and 328 inside the length ofwalls 324. The conduits include a helix section 330 and a distalnon-helix section 329. Appropriate tensioning of pull wire 325 maycompress the distal end 320 in the positive y-axis direction, whileminimizing bending of the helix section 330. Similarly, appropriatetensioning of pull wire 326 may compress distal end 320 in the negativey-axis direction. In some embodiments, the lumen 323 is not concentricwith the sheath 311.

Pull wires 325 and 326 do not necessarily run straight through thelength of sheath 311. Rather, the pull wires 325 and 326 spiral aroundsheath 311 along helix section 330 and run longitudinally straight(i.e., approximately parallel to the longitudinal axis 306) along thedistal non-helix section 329 and any other non-helix section of thesheath 311. The helix section 330 may start and end anywhere along thelength of the sheath 311. Further, the length and pitch of helix section330 may be determined based on desired properties of sheath 311, e.g.,flexibility of the sheath 311 and friction in the helix section 330.

Though the pull wires 325 and 326 are positioned at 180 degrees relativeto each other in FIG. 3C, it should be noted that pull wires of thesheath 311 may be positioned at different angles. For example, threepull wires of a sheath may each be positioned at 120 degrees relative toeach other. In some embodiments, the pull wires are not equally spacedrelative to each other, i.e., without a constant angle offset.

III. B. Helix Sections

FIG. 3D is an isometric view of a helix section 330 of the sheath 311 ofthe endoscope 118 according to one embodiment. FIG. 3D shows only onepull wire for the purpose of distinguishing between the distal non-helixsection 329 and the helix section 330. In some embodiments, the helixsection 330 has a variable pitch.

FIG. 3E is another isometric view of a helix section 330 of a sheath 311of an endoscope 118 according to one embodiment. FIG. 3E shows four pullwires extending along the distal non-helix section 329 and the variablepitch helix section 330.

Helix sections 330 in the sheath 311 and leader 315 of the endoscope 118help a surgical robotic system 100 and/or a user navigate the endoscope118 through non-linear pathways in the anatomy of a patient, e.g.,intestines or the colon. When navigating the non-linear pathways, it isuseful for the endoscope 118 to remain flexible, while still having acontrollable distal section (in both the sheath 311 and the leader 315).Further, it is advantageous to reduce the amount of unwanted bendingalong the endoscope 118. In previous endoscope designs, tensioning thepull wires to manipulate the distal section generated the unwantedbending and torqueing along a length of the endoscope, which may bereferred to as muscling and curve alignment, respectively.

FIG. 3F is a side view of the sheath 311 of the endoscope 118 with ahelix section 330 according to one embodiment. FIGS. 3F-G illustrate howthe helix section 330 helps substantially mitigate muscling and curvealignment. Since the pull wire 340 is spiraled around the length ofhelix section 330, the pull wire 340 radially and symmetricallydistributes a compressive load 335 in multiple directions around thelongitudinal axis 306. Further, bending moments imposed on the endoscope118 are also symmetrically distributed around the longitudinal axis 306,which counterbalances and offsets opposing compressive forces andtensile forces. The distribution of the bending moments results inminimal net bending and rotational forces, creating a low potentialenergy state of the endoscope 118, and thus eliminating or substantiallymitigating muscling and curve alignment.

The pitch of the helix section 330 can affect the friction and thestiffness of the helix section 330. For example, the helix section 330may be shorter to allow for a longer distal non-helix section 329,resulting in less friction and/or stiffness of the helix section 330.

FIG. 3G is another view of the sheath 311 of the endoscope 118 shown inFIG. 3F according to one embodiment. Compared to the distal non-helixsection 329 shown in FIG. 3F, the distal non-helix section 329 shown inFIG. 3G is deflected at a greater angle.

III. C. Endoscope Leader

FIG. 3H is a cross sectional side view of the leader 315 of theendoscope 118 according to one embodiment. The leader 315 includes atleast one working channel 343 and pull wires 344 and 345 running throughconduits 341 and 342, respectively, along the length of the walls 348.The pull wires 344 and 345 and conduits 341 and 342 are substantiallythe same as the pull wires 325 and 326 and the conduits 327 and 328 inFIG. 3C, respectively. For example, the pull wires 344 and 345 may havea helix section that helps mitigate muscling and curve alignment of theleader 315, similar to the sheath 311 as previously described.

FIG. 3I is a cross sectional isometric view of the leader 315 of theendoscope 118 shown in FIG. 3H according to one embodiment. The leader315 includes an imaging device 349 (e.g., charge-coupled device (CCD) orcomplementary metal-oxide semiconductor (CMOS) camera, imaging fiberbundle, etc.), light sources 350 (e.g., light-emitting diode (LED),optic fiber, etc.), and at least one working channel 343 for othercomponents. For example, other components include camera wires, aninsufflation device, a suction device, electrical wires, fiber optics,an ultrasound transducer, electromagnetic (EM) sensing components, andoptical coherence tomography (OCT) sensing components. In someembodiments, the leader 315 includes a pocket hole to accommodateinsertion of a component into a working channel 343.

III. D. Instrument Device Manipulator

FIG. 4A is an isometric view of an instrument device manipulator 117 ofthe surgical robotic system 100 according to one embodiment. The roboticarm 102 is coupled to the IDM 117 via an articulating interface 401. TheIDM 117 is coupled to the endoscope 118. The articulating interface 401may transfer pneumatic pressure, power signals, control signals, andfeedback signals to and from the robotic arm 102 and the IDM 117. TheIDM 117 may include a gear head, motor, rotary encoder, power circuits,and control circuits. A tool base 403 for receiving control signals fromthe IDM 117 is coupled to the proximal end of the endoscope 118. Basedon the control signals, the IDM 117 manipulates the endoscope 118 byactuating output shafts, which are further described below withreference to FIG. 4B.

FIG. 4B is an exploded isometric view of the instrument devicemanipulator shown in FIG. 4A according to one embodiment. In FIG. 4B,the endoscopic 118 has been removed from the IDM 117 to reveal theoutput shafts 405, 406, 407, and 408.

FIG. 4C is an isometric view of an independent drive mechanism of theinstrument device manipulator 117 shown in FIG. 4A according to oneembodiment. The independent drive mechanism can tighten or loosen thepull wires 421, 422, 423, and 424 (e.g., independently from each other)of an endoscope by rotating the output shafts 405, 406, 407, and 408 ofthe IDM 117, respectively. Just as the output shafts 405, 406, 407, and408 transfer force down pull wires 421, 422, 423, and 424, respectively,through angular motion, the pull wires 421, 422, 423, and 424 transferforce back to the output shafts. The IDM 117 and/or the surgical roboticsystem 100 can measure the transferred force using a sensor, e.g., astrain gauge further described below.

FIG. 4D illustrates a conceptual diagram that shows how forces may bemeasured by a strain gauge 434 of the independent drive mechanism shownin FIG. 4C according to one embodiment. A force 431 may directed awayfrom the output shaft 405 coupled to the motor mount 433 of the motor437. Accordingly, the force 431 results in horizontal displacement ofthe motor mount 433. Further, the strain gauge 434 horizontally coupledto the motor mount 433 experiences strain in the direction of the force431. The strain may be measured as a ratio of the horizontaldisplacement of the tip 435 of strain gauge 434 to the overallhorizontal width 436 of the strain gauge 434.

In some embodiments, the IDM 117 includes additional sensors, e.g.,inclinometers or accelerometers, to determine an orientation of the IDM117. Based on measurements from the additional sensors and/or the straingauge 434, the surgical robotic system 100 can calibrate readings fromthe strain gauge 434 to account for gravitational load effects. Forexample, if the IDM 117 is oriented on a horizontal side of the IDM 117,the weight of certain components of the IDM 117 may cause a strain onthe motor mount 433. Accordingly, without accounting for gravitationalload effects, the strain gauge 434 may measure strain that did notresult from strain on the output shafts.

IV. Calibration Dome

During calibration of the endoscope 118, the surgical robotic system 100measures calibration parameters. The calibration parameters may describea movement of the endoscope 118 (e.g., translational or rotational); ahysteresis in pitch or yaw of the endoscope 118; a stiffness in pitch,yaw, or along the length of the endoscope 118; a compression in pitch oryaw of the endoscope 118; a positive or negative pitch angle of theendoscope 118; a positive or negative yaw angle of the endoscope 118; aroll angle of the endoscope 118; and/or a working length between amechanism (e.g., the reference structure 307) coupled to the proximalend and the distal end of the endoscope 118. The endoscope 118 mayinclude a computer readable tangible medium, e.g., flash memory, tostore the calibration parameters. In some embodiments, the calibrationparameters are stored with a unique identifier of the endoscope 118. Thesurgical robotic system 100, via the calibration module 125, can alsostore the calibration parameters in the calibration store 135 and/orupload the calibration parameters and the unique identifier to a globalcalibration database including information from multiple endoscopes.

The calibration parameters may vary between different endoscopes. Forexample, in response to the same command, one endoscope tip rotates 10degrees in pitch while another endoscope rotates 20 degrees in pitch and1 degree in yaw. Thus, the calibration parameters to compensate fornonlinearities of the responses of the two endoscopes will differ invalue. The calibration parameters can be determined for the sheathand/or leader of an endoscope. In some embodiments, the calibrationparameters for the sheath are different than the calibration parametersfor the leader, e.g., because the sheath and leader have different helixsections. The embodiments disclosed herein provide a method andapparatus for accurately and continuously measuring the endoscope'smotion during a calibration process, for example by measuring atrajectory of the endoscope during calibration. The calibration processis automated using the surgical robotic system 100. Although referenceis made to calibration with imaging, the surgical robotic system 100 mayperform calibration using other data collection methods, e.g., usingmagnetic field sensors and accelerometers.

FIG. 5A illustrates an example calibration setup for arobotically-controlled endoscope 118 according to one embodiment. Theendoscope 118 is oriented so that endoscope's tip 301 is secured withina calibration structure 500 with a visual pattern on a surface of thecalibration structure 500 and visible to an image sensor 505, e.g., acamera, mounted on the tip 301. For example, the visual pattern includescheckered squares. A proximal end of the endoscope 118 is secured to thecalibration structure 500 while the endoscope's body 310 and tip 301 canmove around inside the calibration structure 500.

An actuation device 504, e.g., the IDM 117 shown in FIG. 1, is coupledto the endoscope 118 and may receive signals, e.g., from the commandconsole 200 shown in FIG. 2. The signals may also be referred to ascontrol signals or commands. Based on the signals, the actuation device504 manipulates the endoscope 118 within the calibration structure 500.The signals may indicate an intended trajectory for the endoscope 118.As the endoscope tip 301 moves, the camera 505 records image framesrepresenting the perspectives visible to the endoscope tip 301. Duringan image registration process (further described in Section VI. ImageRegistration), the image registration module 130 can measure translationbetween the recorded image frames as a function of time. Thetranslations correspond to movement of the tip 301 in the pitch and/oryaw axis, and thus can be used to calibrate the surgical roboticssystem's pitch and yaw controls.

FIG. 5B illustrates the endoscope 118 positioned within a calibrationstructure 500 during calibration according to one embodiment. Though thecalibration structure 500 shown in FIG. 5B is a dome, it should be notedthat the calibration structure 500 may be a different type of shape inother embodiments. A proximal end of the endoscope 118 is aligned withthe center 520 of the calibration structure 500 dome. The tip 301 hasbeen deflected to a positive yaw angle 513, of θ radians relative to thelongitudinal axis 306. The positive yaw angle 513 may be relatedgeometrically to the deflection angle 514 of φ radians of the endoscope118 relative to the longitudinal axis 306. In use cases where thecalibration structure 500 is a dome, the deflection angle 514 isdetermined by dividing the geodesic distance 515 along the surface ofcalibration structure 500 by the radius 516 of the calibration structure500. The geodesic distance 515 may be determined using imageregistration (further described in Section VI. Image Registration) todetect translation between images from a recorded image sequence. Theyaw angle 513 (θ) may be calculated based on the deflection angle 514(φ), the radius 516 (R), and the distance 517 (r) from the center 520 tothe yaw angle 513, as shown in the following equation:

$\theta = {\sin^{- 1}{\frac{R\;\sin\;\varphi}{\sqrt{R^{2} + r^{2} - {2R\mspace{11mu} r\;\cos\;\varphi}}}.}}$

In cases when r is much smaller than R, the deflection angle 514 (φ) maybe an accurate approximation of the yaw angle 513 (θ). The distance (r)may be predetermined based on physical measurements of the endoscope tip301, or may be calculated during use of the endoscope 118, for example,by measuring changes in distance to a surface of the calibrationstructure 500 based on detected magnification changes corresponding toimage scaling. The negative yaw, positive pitch, and negative pitchangles can be determined using equations similar to the equation shownabove.

V. Calibration Curves

FIG. 6A shows a plot 601 of measured endoscope actual deflection (e.g.,articulation angle of the endoscope) in pitch and yaw in response to acalibration procedure according to one embodiment. During thecalibration procedure, a surgical robotics system 100 actuates theendoscope 118 shown in FIG. 5A in the negative yaw axis 303, positiveyaw axis 302, negative pitch axis 305, and positive pitch axis 304 asshown in FIG. 3A. The calibration module 125 records, using sensors(e.g., image sensors, accelerometers, gyroscopes, strain gauges, etc.)of the surgical robotic system 100 and/or the endoscope 118, the actualdeflection of the endoscope 118 (e.g., in units of degrees) in each axisas a function of a corresponding command (e.g., provided by the commandconsole 200 shown in FIG. 2) to generate the two curves 602 and 603(also referred to as calibration curves) representing the endoscope'sactual deflection in the pitch axis and yaw axis, respectively. Thecommand (also referred to as the command value) represents a targetdeflection, for example in units of degrees. Due to nonlinearities ofthe endoscope 118, the target deflection of a command does not alwaysmatch the actual deflection shown on the plot 601. The calibrationmodule 125 can store the actual deflection, as well as other dataassociated with the calibration procedure such as the correspondingrange of command values and the unique identifier of the endoscope 118,in the calibration store 135.

The actual deflection of both curves 602 and 603 exhibit local linearityas the command value increases or decreases, as well as nonlinearbehavior. In particular, the forward portion 604 of the curve 602 andbackward portion 605 of the curve 602 is offset by hysteresis 606.Likewise, the forward and backward portions of the curve 603 are alsooffset by a hysteresis. Further, the curves 602 and 603 exhibit a “deadzone” 607 around an actual deflection of zero degrees. In the “deadzone” 607, the endoscope is less sensitive to changes to the commandvalue, e.g., relative to the forward portion 604 and backward portion605, the actual deflection changes less per unit of change to thecommand value. For reference, the dashed lines 608 represent an examplemodel without nonlinearities.

The calibration module 125 generates a fit to account for theendoscope's nonlinear behavior. In one embodiment, the fit is apiecewise linear model. The calibration module 125 uses the data fromthe curves 602 and 603 shown in FIG. 6A to generate the four plotsillustrated in FIGS. 6B, 6C, 6D, and 6E corresponding to increasingtarget deflection in the pitch axis, decreasing target deflection in thepitch axis, increasing target deflection in the yaw axis, and decreasingtarget deflection in the yaw axis, respectively.

FIG. 6B shows a plot 607 of linear curves corresponding to increasingtarget deflection in the pitch axis according to one embodiment. Theplot 607 includes a segment of the forward portion 604 and a segment ofthe backward portion 605 of the curve 602 shown in FIG. 6A correspondingto actual deflection in the pitch axis. Based on the two segments, thecalibration module 125 determines a linear fit 609 corresponding to theincreasing target deflection in the pitch axis. For example, the linearfit 609 is a value of the average (or a weighted average) slope of thetwo segments. Further, the calibration module 125 determines thehysteresis 606 based on the width of the gap between the two segments.The calibration module 125 can store values associated with the linearfit 609 and the hysteresis 606, collectively referred to as thecalibration parameters, in the calibration store 135.

The calibration module 125 uses a similar process to determine thelinear fits and hysteresis for the plots shown in FIGS. 6C-E. FIG. 6Cshows a plot of linear curves corresponding to decreasing targetdeflection in the pitch axis according to one embodiment. FIG. 6D showsa plot of linear curves corresponding to increasing target deflection inthe yaw axis according to one embodiment. FIG. 6E shows a plot of linearcurves corresponding to decreasing target deflection in the yaw axisaccording to one embodiment.

VI. Image Registration

FIG. 7 is a flowchart of a process 700 that may be performed as part ofthe process illustrated in FIG. 8 to determine the movements of theendoscope from a sequence of recorded images according to oneembodiment. A controller of a surgical robotics system, for example, thecontroller 120 of the surgical robotics system 100 shown in FIG. 1, usesthe process 700 to calibrate an endoscope. The process 700 may includedifferent or additional steps than those described in conjunction withFIG. 7 in some embodiments, or perform steps in different orders thanthe order described in conjunction with FIG. 7. Since the controller 120automates the process 700, a user does not have to manually perform acalibration procedure to use the surgical robotic system 100.

The image registration module 130 of the surgical robotic system 100shown in FIG. 1 determines calibration parameters of an endoscope tipbased on changes in properties of a sample of images (e.g., grayscale orcolor) captured by an image sensor coupled to the endoscope tip, e.g.,the camera 505 of endoscope 118 shown in FIG. 5A. Because the imagesensor is coupled to the endoscope 118, the image registration module130 assumes that changes between a pair of images of the sample are dueto a shift in perspective of the image sensor corresponding to amovement of the endoscope tip, e.g., translation, rotation, and/orscaling in a pitch or yaw axis.

The image registration module 130 can filter the sample of images, forexample, by removing every other image of the sample to help reduce thetime required to process the sample. In some embodiments, the imageregistration module 130 extracts the sample of images from a videocaptured by the image sensor. Image registration does not require thesource and target images to be subsequent frames of the camera. However,the accuracy of the motion estimated by image registration tends to begreater as the time period between images decreases. Thus, the imageregistration module 130 generates more accurate motion estimates (e.g.,nearly continuous measurement of calibration parameters) by registeringmany images in sequence.

To determine translation movement, the image registration module 130receives 710 a sample of images and analyzes pairs of images of thesample using an optical flow technique. In a pair of images, the imagethat occurs first is referred to as the source image and the image thatoccurs second is referred to as the target image. The order of the firstand second images is arbitrary. Thus, the direction of translation(e.g., moving forward or backward in time) is determined based on whichimage is considered the source and which images is considered thetarget. In one embodiment, each image is a two-dimensional pixel arrayof N pixel values corresponding to light intensities (e.g., forgrayscale images), vectors representing intensities of different colorsof light (e.g., for color images), etc. The image registration module130 can transform the two-dimensional pixel array into a corresponding1-dimensional array with N elements for processing.

The image registration module 130 generates 720 a difference array D andgenerates 730 a gradient array G based on the pair of images. In someembodiments, the image registration module 130 generates a differencearray and gradient array for each pair of images of the sample. Thedifference array D is based on the difference between a pixel value ofthe target image and a corresponding pixel value of the source image.The gradient array G is based on a weighted average of the rate ofchange (e.g., derivative) of a pixel value of the target image and therate of change of a corresponding pixel value of the source image. Inembodiments with a two-dimensional (e.g., x and y dimensions) pixelarray, the rate of change of a pixel in the x-dimension G_(x) is basedon the difference between the pixel and each of two or more adjacentpixels in the x-direction. Similarly, the rate of change of the pixel inthe y-dimension G_(y) is based on the difference between the pixel andeach of two or more adjacent pixels in the y-direction. The gradientarray may be a weighted average of the rates of change in the x and ydimensions, e.g., equally weighted. The image registration module 130can decompose the 2D gradient array into two sub-arrays, G_(x) andG_(y), corresponding to partial derivatives in the x and y directions,respectively. Accordingly, the image registration module 130 representsG as an N×2 matrix: G=(G_(x) G_(y)), where G_(x) and G_(y) each includeN components.

The image registration module 130 determines a set of calibrationparameters represented by the vector p. In some embodiments, the imageregistration module 130 determines a set of calibration parameters foreach image pair based on the gradient array G and difference array D.The image registration module 130 can repeat the steps 720-750 of theprocess 700 for multiple pairs of images of the sample. Thus, the imageregistration module 130 generates a set of calibration parameterscorresponding to each processed pair of images.

The vector p includes a set of model parameters (e.g., representingdifferent types of movement of the endoscope tip) and can be modeled assatisfying a linear equation of the form: Ap=v, where A=(G_(x) G_(y)),v=D. The image registration module 130 can use a least squaresregression analysis to estimate that p=(A^(T)A)⁻¹A^(T)v, where A^(T)represents the transpose of A and (A^(T)A)⁻¹ represents the inverse ofthe product of A^(T) with A. Thus, the image registration module 130determines that

${p = \begin{pmatrix}t_{x} \\t_{y}\end{pmatrix}},$where t_(x) and t_(y) represent translational movement of the endoscopetip in the x and y dimensions, respectively.

The image registration module 130 can also determine rotational movementof the endoscope tip corresponding to an angle of rotation, θ. Forexample, the image registration module 130 may represent p as atwo-dimensional vector of the form

${= \begin{pmatrix}a \\b\end{pmatrix}},$where b represents the sine of the angle of rotation (e.g., θ) and arepresents the square of the cosine of the angle of rotation (e.g., θ)minus 1. Note that for small angles of rotation, b≈θ and a will be small(a≈−θ²). The image registration module 130 determines a matrixA=(G_(x)r_(x)+G_(y)r_(y) G_(x)r_(y)−G_(y)r_(x)), where the vectors r_(x)and r_(y), denote the positions of a given pixel relative to the centerof rotation. The image registration module 130 determines p to estimatethe angle of rotation by solving the equation Ap=v. In cases where asource image has been scaled, for example, due to a change in distancefrom the source image to the target image, the image registration module130 determines the scale factors based on the equations:s²=(a+1)²+b{circumflex over ( )}2, and b/s=sin θ≈θ.

The image registration module 130 can generate a matrix A that combinesthe translational and rotational movement components as shown below:

${A = {{\begin{pmatrix}{{G_{x}r_{x}} + {G_{y}r_{y}}} & {{G_{x}r_{y}} - {G_{y}r_{x}}} & G_{x} & G_{y}\end{pmatrix}\mspace{14mu}{and}\mspace{14mu} p} = \begin{pmatrix}a \\b \\t_{x} \\t_{y}\end{pmatrix}}};$

The image registration module 130 can transform A using an arbitrarymatrix of the form

$\begin{pmatrix}{1 + a} & b \\c & {1 + d}\end{pmatrix},$resulting in:

$A = {{\begin{pmatrix}{G_{x}r_{x}} & {G_{x}r_{y}} & {G_{y}r_{x}} & {G_{y}r_{y}}\end{pmatrix}\mspace{14mu}{and}\mspace{14mu} p} = {\begin{pmatrix}a \\b \\c \\d\end{pmatrix}.}}$

The image registration module 130 uses the calibration parameters p togenerate a sequence of transforms T_(i), where T_(i) represents atransform from the ith to the (i+1)th image of a sample. The vectorp_(n) includes the calibration parameters for the nth image, and thevector p_(n+1)=T_(n)p_(n) includes the calibration parameters for the(n+1)th image. T_(i) may indicate motion in one or more axis betweenimages.

To obtain the calibration parameters p as a function of image number,the image registration module 130 applies the transforms sequentially toa starting vector p₁, so that p of an arbitrary image n is:

$p_{n} = {\left( {\prod\limits_{i = 1}^{n - 1}T_{i}} \right){p_{1}.}}$

Generally, T_(i) does not commute, so each T_(i) is applied in orderstarting with T₁. The sequential measurements of p_(i) may represent atrajectory, for example, a movement from an initial position ororientation p₁ continuously through a series of positions ororientations to a final position or orientation p_(n). Thus, the imageregistration module 130 can determine an unknown p for an arbitraryimage using a known p for a starting image and applying a sequence ofknown transformations using the equation shown above. The calibrationparameters may include measurements in units of pixels. The imageregistration module 130 can convert the units using conversion factors.For example, an object of known size in millimeters in an image ismeasured in pixels to determine a conversion factor from millimeters topixels.

In some embodiments, the image sensor includes one or more colorchannels, e.g., three color channels corresponding to red, green, andblue (RGB) light colors. Since each color channel may be sensitive todifferent colors, more accurate measurements during a calibrationprocess may be obtained using a multi-colored target. For example, thesurface of a calibration structure such as the dome calibrationstructure 500 shown in FIG. 5A includes a patterned surface of multiplecolors, e.g., an alternating pattern of red and green checkered squares.The image registration module 130 represents color images using anadditional dimension. Further, the image registration module 130 canindependently determine the matrices A_(i) and vectors v_(i) for eachcolor channel i in the same manner as A and v as described above. Theimage registration module 130 may concatenate A_(i) and v_(i) intomatrices A and v:

${A = \begin{pmatrix}A_{1} \\\vdots \\A_{M}\end{pmatrix}};{v = {\begin{pmatrix}v_{1} \\\vdots \\v_{M}\end{pmatrix}.}}$

The calibration parameters may be more sensitive to certain colorchannels. For example, in RGB images of a calibration structure surfacethat includes red and green colored squares, the calibration parametersare more sensitive to the red and green channels than the blue channel.Image data from the blue channel may predominantly represent noise,while image data from the red and green channels may represent signal.The image registration module 130 can adjust the sensitivity of a colorchannel by applying 740 weights to the difference array and/or thegradient array. For example, for each color channel, the imageregistration module 130 multiplies each matrix A_(i) and vector v_(i) byan independently-variable vector of weighting parameters w_(i) prior toconcatenation:

${A = \begin{pmatrix}{w_{1}A_{1}} \\\vdots \\{w_{M}A_{M}}\end{pmatrix}};{v = {\begin{pmatrix}{w_{1}v_{1}} \\\vdots \\{w_{M}v_{M}}\end{pmatrix}.}}$

The image registration module 130 generates 750 a set of calibrationparameters based on the difference array and the gradient array. Thecalibration parameters corresponding to the weighted versions of A and vare more dependent on the color channels with larger weight than thosewith smaller weight. For example, to produce calibration parameters thatrespond equally strong to red and green color channels, but weak to theblue color channel, the weight for the red and green channels is 1, andthe weight for the blue channel is 0.05. The weighting parameters may beadjusted to account for a wide range of experimental variables,including camera sensitivity, target pattern color of a calibrationstructure, or the color of illuminating light. In some embodiments, theimage registration module 130 further customizes the sensitivity of theset of calibration parameters by using other types of weighting methods,e.g., nonlinear weighting functions or weighting functions based onvariables such as pixel location.

VII. Process Flows

FIG. 8 is a flowchart of a process 800 for automated calibration of anendoscope according to one embodiment. The process 800 may includedifferent or additional steps than those described in conjunction withFIG. 8 in some embodiments, or perform steps in different orders thanthe order described in conjunction with FIG. 8. Since the controller 120is capable of automating the process 800, a user does not have tomanually perform a calibration procedure to use the surgical roboticsystem 100. Automated calibration is advantageous, e.g., because theprocess reduces the time required to calibrate an endoscope.

The calibration module 125 of the controller 120 provides 810 one ormore commands from the surgical robotic system 100 to an actuator, forexample, the IDM 117 shown in FIG. 1, to move the endoscope 118 for acalibration procedure. The endoscope may be positioned in a calibrationstructure (e.g., calibration structure 500 shown in FIG. 5A) during thecalibration procedure. Based on the commands, the IDM 117 moves theendoscope in a translational and/or rotational motion in one or moreaxis, e.g., the positive yaw axis 302, negative yaw axis 303, positivepitch axis 304, negative pitch axis 305, or roll axis 306 shown in FIG.3A.

The calibration module 125 receives 820 images captured using an imagesensor on the tip (e.g., tip 301 shown in FIG. 3A) of the endoscope 118.The images may include a sample of one or more adjacent images (i.e., insequence) or non-adjacent images. The images correspond to a movement ofthe endoscope 118. For example, the calibration module 125 provides thecommands to the IDM 117 in step 710 and simultaneously (or soonafterwards) provides a coordinated command to the endoscope 118 tocapture the images using the image sensor.

The image registration module 130 of the controller 120 generates 830 afirst set of calibration parameters by performing image registration onthe captured images, as previously described in Section VI. ImageRegistration. The first set of calibration parameters can include valuesrepresenting translational and/or rotational movement of the endoscopetip 301 in one or more axis, e.g., pitch and/or yaw.

The calibration module 125 generates 840 a model of the endoscope'smovements based on the captured images. In an example use case, duringthe step 710, the IDM 117 moves the endoscope forward and backward inboth a pitch axis and a yaw axis. The resulting model can be illustratedby the calibration curves 602 and 603 as shown in plot 601 in FIG. 6Arepresenting the endoscope's motion in the pitch and yaw axis,respectively. Each calibration curve may be associated with an axis ofmotion of the endoscope 118.

The calibration module 125 generates 850 a second set of calibrationparameters based on the calibration curves. Following in the sameexample use case, the calibration module 125 uses curve fitting todetermine values for a slope, hysteresis, and/or “dead zone” to includein the second set of calibration parameters. For example, the values maybe based on the calibration curves 602 and 603, as shown in the plots inFIGS. 6A-E. The plots in FIGS. 6B-E each represent a linear portion of acalibration curve of the plot 601 corresponding to one of: increasingarticulation angle in the pitch axis, decreasing articulation angle inthe pitch axis, increasing articulation angle in the yaw axis, anddecreasing articulation angle in the yaw axis.

The calibration module 125 stores 860 the first set of calibrationparameters and/or the second set of calibration parameters in thecalibration store 135 or any other database accessible to the surgicalrobotic system 100. The calibration module 125 may store the sets ofcalibration parameters with a unique identifier associated with thegiven endoscope 118. In some embodiments, the calibration store 135includes a lookup table that stores calibration parameters mapped tounique identifiers. Thus, the calibration module 125 can retrieve acalibration parameters associated with a given endoscope using thelookup table with an input unique identifier. In some embodiments, thecalibration module 125 stores the sets of calibration parameters with atype of command (e.g., translation or rotation in a given axis)corresponding to the commands used to move the endoscope 118 in step810.

FIG. 9 is a flowchart of a process 900 for controlling an endoscopeusing calibration parameters, according to one embodiment. The process900 may include different or additional steps than those described inconjunction with FIG. 9 in some embodiments, or perform steps indifferent orders than the order described in conjunction with FIG. 9.The command console, such as command console 200, may use the process900 in the velocity mode or position control mode previously describedin Section II. Command Console.

The command console 200 receives 910 a command to move the endoscope 118using the surgical robotic system 100, e.g., using the robotic arms 102and the IDM 117 shown in FIG. 1. The command may cause the endoscope 118and the tip 301 of the endoscope (as shown in FIG. 3A), to translate orrotate in one or more axis. The command can be received from a user ofthe surgical robotic system 100 via the control modules 203 and 204shown in FIG. 2. In other embodiments, commands can be received from aprocessor or database of the surgical robotic system 100, e.g., apre-programmed routine of motion commands associated with a surgicalprocedure.

The command console 200 receives 920 calibration parameters associatedwith the endoscope 118 or associated with a type of the command. Thecalibration parameters may include calibration parameters generatedusing the process 700 shown in FIG. 7 using image registration (e.g., bygenerating difference arrays and gradient arrays). The calibrationparameters may also include calibration parameters generated using theprocess 800 shown in FIG. 8 using calibration curve fitting. The commandconsole 200 generates 930 an adjusted command based on the command andthe calibration parameters. The adjusted commands account for nonlinearbehavior (e.g., corresponding to translational and/or rotational motionin one or more axis) of the endoscope 118 using the calibrationparameters. In an example use case, the command describes atranslational motion of 10 degrees in the positive pitch axis and 20degrees in the positive yaw axis. Due to nonlinearities of the endoscope118, the endoscope 118 instead translates 5 degrees in the positivepitch axis and 30 degrees in the positive yaw axis based on the command.The calibration parameters indicate that the translation in the positivepitch axis undershoots by 5 degrees and the translation in the positiveyaw axis overshoots by 10 degrees. Thus, the adjusted command describesa translational motion of 15 degrees in the positive pitch axis and 10degrees in the positive yaw axis to compensate for the nonlinearities.

In embodiments where the calibration parameters include both a first setof calibration parameters (generated using image registration) and asecond set of calibration parameters (generated using calibration curvefitting), the command console 200 generates the adjusted command bycombining different types of calibration parameters. For example,calibration parameters generated using image registration includestranslations and rotations. In addition, calibration parametersgenerated using the calibration curve fitting process includes slopesand hysteresis. The command console 200 can first apply a translationfrom the first set to modify the command and then apply a hysteresisfrom the second set to modify the command again, resulting in the finaladjusted command. In other embodiments, the command console 200 appliesany number of different calibration parameters from one or both sets inany particular order.

The command console 200 provides 940 the adjusted command to thesurgical robotic system 100 to move the endoscope 118.

The command console 200 receives 950 endoscope information describingthe position or orientation of the endoscope 118, e.g., in response tothe surgical robotic system 100 to moving the endoscope 118 based on theadjusted command. The endoscope information may be captured by sensors(e.g., accelerometers, gyroscopes, etc.) of the robotic arms 102 orother sensors such as the strain gauge 434 of the IDM 117 shown in FIG.4D.

The command console 200 stores 960 the adjusted command and theendoscope information in the calibration store 135 of the controller 120shown in FIG. 1 or any other database accessible to the surgical roboticsystem 100. The command console 200 can use the endoscope information todetermine whether the calibration parameters correctly accounted fornonlinearities of the endoscope 118.

Continuing with the same example use case, the endoscope informationindicates that the endoscope 118 translated 9 degrees in the positivepitch axis and 21 degrees in the positive yaw axis based on the adjustedcommand. Since the original command corresponded to translations of 10degrees in the positive pitch axis and 20 degrees in the positive yawaxis, the endoscope 118 still undershot by 1 degree in the positivepitch axis and overshot by 1 degree in the positive yaw axis. Thus, thecommand console 200 determines that the calibration parameters did notfully account for the endoscope's nonlinearities.

The command console 200 can use the endoscope information to implementfeedback control of the endoscope 118. Particularly, the command console200 can modify the adjusted command based on the endoscope information.For example, since the endoscope information indicated that endoscope118 undershot by 1 degree in the positive pitch axis, the commandconsole 200 modifies the adjusted command to translate the endoscope 118by an additional 1 degree in the positive pitch axis to compensate forthe difference. The command console 200 can upload the endoscopeinformation and the adjusted command to a global calibration databaseincluding aggregate information from multiple endoscopes and surgicalrobotic systems.

VIII. Endolumenal Procedures

The surgical robotic system 100 can use stored calibration parameters toperform surgical procedures on a patient. FIGS. 10A-C and FIGS. 11A-Cillustrate example surgical procedures using an endoscope, e.g.,endoscope 118 shown in FIG. 3A. The calibration parameters allow thesurgical robotic system 100 to more accurately navigate the endoscopeinside the patient to perform the surgical procedures.

FIG. 10A illustrates the distal end of the endoscope 118 within ananatomical lumen 1002 according to one embodiment. The endoscope 118includes a sheath 311 and navigates through the anatomical lumen 1002inside a patient toward an operative site 1003 for a surgical procedure.

FIG. 10B illustrates the endoscope 118 shown in FIG. 10A in use at theoperative site 1003 according to one embodiment. After reaching theoperative site 1003, the endoscope 118 extends a distal leader section317, longitudinally aligned with the sheath 311, in the direction markedby arrow 1005. The endoscope can also articulate the distal leadersection 317 to direct surgical tools toward the operative site 1003.

FIG. 10C illustrates the endoscope 118 shown in FIG. 10B with anaspiration needle 1007 according to one embodiment. In cases where theoperative site 1003 includes a lesion for biopsy, the distal leadersection 317 articulates in the direction marked by arrow 1006 to conveythe aspiration needle 1007 to target the lesion.

FIG. 11A illustrates an endoscope 118 coupled to a distal leader section317 within an anatomical lumen 1104 according to one embodiment. Theendoscope 118, including a sheath 311, distal leader section 317, andforceps 1103, navigates through the anatomical lumen 1104 toward anoperative site 1106. In some embodiments, the distal leader section 317is retracted within the sheath 311. The construction, composition,capabilities, and use of distal leader section 317, which may also bereferred to as a flexure section, are disclosed in U.S. patentapplication Ser. No. 14/201,610, filed Mar. 7, 2014, and U.S. patentapplication Ser. No. 14/479,095, filed Sep. 5, 2014, the entire contentsof which are incorporated by reference.

The distal leader section 317 can be deployed through a working channelthat is off-axis (neutral axis) of the sheath 311, which allows thedistal leader section 317 to operate without obscuring an image sensor(not shown in FIG. 11A) coupled to the end of the sheath 311 (or anyother location of the endoscope 118). This arrangement allows the imagesensor to capture images inside the anatomical lumen 1104 while theendoscope 118 articulates the distal leader section 317 and keeps thesheath 311 stationary.

FIG. 11B illustrates the endoscope shown in FIG. 11A with the forceps1103 in use at the operative site 1106 according to one embodiment. Theendoscope 118 articulates the distal leader section 317 in the directionof arrow 1105 to orient the forceps 1103 toward the operative site 1106.The forceps 1103 takes a biopsy of anatomical tissue at the operativesite 1106, e.g., for intraoperative evaluation of the patient. In otherembodiments, the endoscope 118 includes a surgical tool different thanthe forceps 1103, for example graspers, scalpels, needles, probes, orlaser devices, which is further described below. The endoscope 118 cansubstitute the surgical tool intra-operatively to perform multiplefunctions in a single surgical procedure inside an anatomical lumen.

FIG. 11C illustrates the endoscope 118 shown in FIG. 11A with a laserdevice 1107 in use at an operative site according to one embodiment. Thelaser device 1107 emits laser radiation 1108 at the operative site 1106for tissue ablation, drilling, cutting, piercing, debriding, cutting, oraccessing non-superficial tissue.

IX. Alternative Considerations

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs throughthe disclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various modifications,changes and variations, which will be apparent to those skilled in theart, may be made in the arrangement, operation and details of the methodand apparatus disclosed herein without departing from the spirit andscope defined in the appended claims.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context unlessotherwise explicitly stated.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

What is claimed is:
 1. A method for automated calibration of anendoscope of a surgical robotic system, comprising: providing a commandto move the endoscope using the surgical robotic system to cause motionof the endoscope; receiving a plurality of images captured by an imagesensor associated with a distal end of the endoscope during the motionof the endoscope; generating, for a pair of images of the plurality ofimages, a gradient array based on rates of change in pixel valuesbetween a first image of the pair of images and a second image of thepair of images; generating a first set of calibration parameters basedat least in part on the gradient array; and storing the first set ofcalibration parameters in association with the endoscope.
 2. The methodof claim 1, further comprising generating, for the pair of images of theplurality of images, a difference array based on differences in pixelvalues between the first image of the pair of images and the secondimage of the pair of images, wherein the first set of calibrationparameters is based at least in part on the difference array.
 3. Themethod of claim 2, wherein: the image sensor includes a plurality ofcolor channels; and the method further comprises applying a weightedaverage to the difference array and the gradient array based on asensitivity of each of the plurality of color channels.
 4. The method ofclaim 1, wherein said storing the first set of calibration parametersinvolves storing a unique identifier of the endoscope associated withthe first set of calibration parameters in a database storinginformation associated with multiple endoscopes.
 5. The method of claim1, further comprising: receiving movement information indicating actualmotion of the endoscope in response at least to the command; andgenerating a calibration curve indicating a relationship between theactual motion and the command.
 6. The method of claim 5, wherein themovement information includes at least: a first segment describingmotion of the endoscope in a forward direction along an axis; and asecond segment describing motion of the endoscope in a backwarddirection along the axis.
 7. The method of claim 6, further comprising:determining one or more slope parameters associated with the calibrationcurve by averaging a first rate of change of position of the endoscopebased on the first segment and a second rate of change of position ofthe endoscope based on the second segment; and determining one or morehysteresis parameters based at least in part on a difference between thefirst segment and the second segment.
 8. The method of claim 5, furthercomprising generating a second set of calibration parameters based atleast in part on the calibration curve.
 9. The method of claim 8,wherein the second set of calibration parameters includes slope andhysteresis parameters.
 10. The method of claim 8, wherein the second setof calibration parameters is generated using curve fitting of thecalibration curve.
 11. The method of claim 5, wherein the calibrationcurve represents actual yaw and pitch deflection of the endoscope as afunction of a corresponding command.
 12. The method of claim 1, whereinthe pair of images are non-sequential.
 13. The method of claim 1,wherein the pair of images are images of a surface of a calibrationstructure.
 14. The method of claim 13, wherein the calibration structureis a dome structure.